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ABSTRACT 

Since acceptable writing is essential to success in 
job training prograas and inaAny entry-level jobs, a writing sample 
was inclttded in the Training and Ea^loyaunt Prerequisites Survey, a 
Multiple-choice test about skills like mechanics, usage, and 
spelling. The two writing presets asked students to give directions 
for finding a location in a school building (Form A) and to compose a 
business letter that requests adjustment of an order (Form B). The 
scoring criteria for Form A were as follows: (1) submrts scorable 
sample, (2) includes important information about direction of 
movement, (3) includes critical information about features a^ng the 
way, (4) writes directions in logical order, (5) uses pi^ecise 
language, (6) uses correct gr«nmar and complete sentences, (7) 
capitalizes and punctuates correctly, and (8) spells correctly. 
Criteria for Form B took into consideration how well the student 
stated the problem and the desired course of action, used precise 
language, used language appropriate for a business letter, excli^ed 
unrelated sentences, used correct grammar and complete sentences, 
capitalised and punctuated correctly, and spelled correctly. Field 
tests of the two forms yielded Svme changes in criteria and the 
lacguage of the prompts. Both writing sample assessments will be 
subjected to further revision based on additional experience with 
them. (HOD) 
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WRITING A WRITING ASSESSMENT: 
AS SIMPLE AS IT SEEMS 

Adrienne Escoe { 



SAYING WHAT YOU WANT TO SAY ISN'T 



ABSTRACT 

The paper presents an ethnographic account that traces fhe 
creation of a writing 5«nple hisses sn«nt and provides guidance for 
those who design and develop -assessments of students' writing. 
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WRITING A WRITIMC; ASSESSMENT: SAYING WHAT YOU'WANT Tt) SAY ISN'T 

AS SIMPLE AS IT. SEEMS • 

Adrienne Escoe^ 

School districts have begun. In recant years, to devote more resources 
(e.g., staff, materlafs) to wrftlng instruction. Many have developed pro- 
grams of Instruction centered on discourse products. Typically, students 
are given a saddle prcmpt, they write a response, and then have theJr 
writing evaluated. We've seen collections of proRipts used as the foun-, 
datlcfn of Instruction; that Is, a new prowpt Is assigned each week, every 
two weeks, or some other time Interval (e.g.. F-u 11 erton Union High School 
District Competency Review: Paragraph Review Booklet. 1981). Usually 
teachers evaluate ("grade") students' papers, sometimes p'ace comments 
on them, return the papers, and discuss a handful or so. 

t 

^ ' Eviiluatlon criteria almost always Include grairmar and usage, spelling, 
capitallzatron. punctuation-all , -elements of form. Every&y- teachers . 
parents, employers-w^nts good *<rltlng form. And most evaluations of 
form are easy to come by. It's npt too difficult to set standards (I.e., 
criteria) for correct form: either the first v.6rd of a sentence Is 
caRltalizcd or It's not; either the period's at the end or it's. not. 
Establishing standards for content , however. Is a different story. 



^Students and teachers from the Los Angeles Unified School District 

and Garden Grove . Un I f i ed| School District participated In the study, and 
the following SWRL staff were involved In various aspects Including creating 
the original versions of the writing prompts, trying them out informally, 
serving as tield-test readers and liaison with the teachers, .and processing 
dat^ Barry Bachelor. Bruce Cronnell. Larry Gentry, Ann Hunws. Joe Law or, 
Shirley Paddlt, Jan Perkins, Gall Reneau, Roger Scott, Karen Smitu, DSvId 
Snw, and Nancy Yaraani 



Characteristics of content, e.g., staying to the main ,toplc, are 
included In some criteria used for assessing written drscoursc; more often,, 
the standards are broad and ratiVer vague, with the only c^it«r^oh often- 
being whether the writer has responded directly to the proB^st given. 

Writing a good wrllflng pron^it Is not so easy; but harder, s-tlll is 

-*■■-• 

seeing to J t that the criteria arid the prosit are written to suit each 
other we! reestablishing criteria of form and content, and then checlcing 
empiRlcally to make sure that students respond to the prompt and that 
teachers evaluate the wfltln^ samples as Intended. That's the only way 
thus far we know how to .dcr.Lrmlne whether the criteria and the prompt are 
gooif. I.e., the prompt U productfvc ami th^ sample is Judgeable. And 
without those characteristics a,writlng sample is not worth the paper it's 
written on. 

This paper Is an ethnographic account of the creation of a writing 
sample assessment and Is -a guide for those who design and develop 
assessments of students' writing. ^ 

To include a Writing Sample . ^ ^ . . 

Thctxicaston for the assessment was the development of Instrwnentation 
to ensure that individuals who are popularly term«l* "renedial" and "hafd- 
to-emp.loy" acquire the prerequisites for success in training and employment, 
Through surveys and analyses of Instruction and of job- training prospects 
for these individuals,, we determined that clear, correct writing wa^ a 
desired and teachable accomplishment. And one way to estimate students' 
writing accomplishments is to have them choose correct/rcsivonses to 



muUi pie-choice Items about skills like mechanics, usage, and spelling. 

Ue Included Items »f this sort on the TEPS (the acronym fdr Training 

% . ' - 

and Employment Prerequisites Survey).^ But raul tlple-cholce Items do not 

sfescflbe productive writing, I .e. written discourse. Our thinking was 

that high-risk students (those headed for uncmptoyment, unstable employ- 

ment, or marginal envloyraent) needed to be able to produce acceptable 

I 

wfUlng to enhance\thelr chances of Sidestepping a predictable fate; 
acceptable writing is essential tc success In job-training programs and 
Tiany entry-level jobs. Thereupon, we decided to Include writing - 

• * 

samples In the TEPS. 

» 

Writing the TEPS Writing Sample Assessments 

The next stage was to'ask questions: What kind x>f writing would 
entry-Jevel employees or workers-ln-tralning need to do? What kind df^ 
writing had they practiced In school? What kind of writing would "get 
at" the skills we wanted to describe? For answers we looked to SWRL 
research/on writing. Several documents were particularly helpful (Escoe, 
1982a. b; Gentry. 1982; Him«s, I98O; Humes, Cronnell, Law lor. Gentry, & 
Fleker. 1980;- Hack. 1982). 

Training and employment dataT Indicate that clear, accurate comnuni- 
cation is one of employers' top priorities. And service to the public 
is' involved In the majdrlty of entry-level positions. So it was a short 
/step to specify a writin^j s*. whose primary emphasis was to comrminicat^ 

a message clearly and serve the needs of othe/ people-. But other con- 
" siderations were equally important, among them, topic familiarity, 
writing type practiced In school, and simple vocabulary. 



To dmnstrate fairly sttidertts' aqcompllShmOTts," It Mas essential , 

.9 
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chat the Mritlng sati^le asses^mt consist only of tasks that would be 
familiar, that Is. practiced In school. One .skill artfo emphdsUed In 
remedial reading and English ciasses, the data indicate. Is sequential 
information. Stuitents often read sequentially-arranged te^ and answer 

* • • * / * * 

questions about the order of events. Other frequmt ifctlvit,ies In tliese 

ft 

classes are writing statements. In logical sequence «nd developing a para- 
graph the sane way,. Another cohrioo focus of instruction Is Interpretlhg 
and writing business correspondence. Typically, programs stress written 
conventions, such as forwat "bftd appropriate l«Higuage, though of the two 
examples, letter" fonaat is ^nore coawonly taught. But since multiple- 
choice items that assessed knowledge of letter format were already 
designed Into the TEPS. and buslness-1 Ike writing Is so Injportant a part 
of employment sutcess. appropriate language in a business letter was 
cfhosen for inclusion In tte TEPS writing sample assessment. 

Next, the vocabulary and syntax of the assessment prompt and the 
directions for stilts had to be famlll?r and simple. For vocabulary, 
I referred to' the EDL iJore list, grades three to seven for Form A, 
grades six to nine for B. In that way. we had some assurance that 
students had oiciiun' sred words In instructional materials or at least 
in widely iiscd test materials such as the Wllfomla Test of, Basic Skills. 
Then, to effect syntactic sin^licity^ sentences were written to be 
straightforward. I.e., without troublesone transformations and Inclusions. 
The idea was to avoid any characteristic of the assessment likely to pre- 
sent reading difficulties that would interfere with wrft>ftg performance. 



To assess m st^^^nt's wrltjntl tjerfenaanee fairly ^^tlw prxw^it and 
the directions must clarify what is expected. . For exan^e, If correct 
' spell Ing ls 6ne of the sklH^ to be assessed," students should be made 
-vMre of that factuotherwise, so«e. students .way not attend to spelling, 

» I 

under "the errbnodlis assumption that spelling "doesn't count." 

At this point,- three additional ^eneraj criteria were established 
for developing k *irl ting- sample -assessment ^ produces I) easily-scored - 
writing,* 2) diagnostic Information, and J) re 1 fable. Information. But 
' measuring ^he assessment against these three had to vtait until the 
previously descrlbe^'crfterTa were work^ out. To sum up, the general 
criteria and, correspondingly,, the specific critcrla-for the TEPS *#ere 
-the ones ihown. on the following page. 

The next step i<i writing the writing sample pron^t Involved creativity 
more than any th I ng else. Thumbing through reading and English workbooks . 
and published and district assessments, and SWL technical docurodhts, but 
mostly j^st bri^lnstorralng, we listed, evaliwted, and ultimately rejec ed 
s'cores'-of topics for the writing swlsa. .Two topics, however, tentatively 
seemerf to conform to all the established criteria: gIvKng dl^ctlons for 
finding a location Iri^'a school bul l^lng»(Fonn A) and cos^JosIng the body 
of a buslness'letter"that requests, adjustment of an order (Form B). Both 
topics seemed likely to elicit a discourse product whose primary en^hasls 
wouK.be to coffinunlcate a message clearly and serve a person 'speeds'. 
Oiri*ctlons for getting to a room In a school building had to-be clear, 
accurate, and given In a logical sequence to be helpful to the hypothetical 
person who followed theo. the business letter had to be written clearly 
with desired action' stated precisely so that the Iroaqinary recipient 



Wrtt.In9 Sample Prompt 



TCPS Criteria 



Seiwra! Criteria 

1. SkuU valued fot- training 
and entry-lav«l enploynisnt 

-2. Skill areas practiced In 

SClkOOl 

3. Fomat practiced lii' $cNx>V 



k. ' S1iif>lc. fawlllar vocabulary 



5. Simple, straightforward 
syntax ' '* • 

6. Precise statement of 
requ I rodents 

7: Produces easily-scored 
writing 



Form A 



Form 6 



8. Produces diagnostic 
Information 



9. Produces reliable Information 



CI ear ..accurate conmunlcatlon 
Public service activity 



Seqimntlal 
Information 

Wrl*-ljig. In 
seqittnce of 
occurrence 

Core vrcabulary 
grade le^^ls. 
3 throiqh 7 



Business 
correspondence 

I* 

Writing a' 
business 
lettef 

Cqre vocabula^'^ 
gra^ levels 
6 through 9 



Complex sentence constructions 
avoided « 

All crltei^ for evaluiftlon , 
spelled out to»exaro1 nces 

Cerarates scoring crUterIa that, 
require little or no training of 
evaluators, and that pertain, 
specifically to the prompt 

Generates scoring criteria that' 
separately pertain to features of- 
conteht and form 

Readers tend, to agree in their 
assessments 
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Mpuld be able respond -sdUsfdCtorily to the request. I.e.,. make j 
satisfactory adjustnrnt t6 order. Each topic, further, represented 
'pr^l^tlced skin* and^sklU fbrmats:* wrflfng events In sequence, and 
writing a bu<ifness 1ett;er. Vocabulary of the writing prompt could be 
kept fwniliar and s^rnple* -for the Form A prompt, no tcj^hnical terms 
Mould be rteeded: and stwtents should be qiilte familiar with the names- 
pf features, of a school btilMIng and slrsple directional terms (e.g.-, 

■ ; ,. » - 

left, eight, north, sou^h). '.For Form B, some familiarity with the 
business world' would be n^ited, so we looked long atd hard for a type 
of business tha^most stuiNMits wcHild kiKM about, and om for which, 
knoiledge of technical %em^was not required. After countless attempts, 
we xcroed In business that handled office supplies. Even If students 

*9 ^ 

were coR^Ietely Ignorant of the office supply business, they sljould be 
able^to respond to this topic because o^ school and home uses of office 
supplies and because of the open nature of the prwrpt Itself (which'will 
be discussed latef). ftoreovcr, since t^ils topic was a candidate for the 
Form B assessment, our thinking m that the more advanced students 
(Form B examinees) would^tend to be closer 'to entering the job market, 
And, realistically. If they had no knowledge at all of cownon materials 
used In an office, they probably wouldn't be ready for entry-level jobs 
or job training^ (even the slightest knowledge of materials would get 
^students by with the prompt as it c«ne to b«« written). The /jext con- 
sideration In writing the prompt was syntax. Nothing more than careful 
construction was required here— no*" complicated transformations, no 
intrusions, no very*^ longx sentences. Next, students had to be told pre- 
cisclywhat teachers would be ^'lookinq for'* In their writing. This 
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crltvrton was difficult to meet; only through a series of field tests 

f 

thet IncluM rcMorded prmpts Msre we able to advise stt^ents on what 
to Inclifde In their writing without having them produce virtually literal 
recasting of the prtmpts. Our exiwrlence In developing the TEPS writing' 
s»i^I« prcmpts oHifiraed the observation, by Humes et al. that piloting with 
target populations Is essential (I980). 

. Llliewlte, the seventh general criterion can be addressed only by 
field testing: rmders (Ideally those who represent readers who will 
function Under •ctual-^ondi ticHis of TEPS administration and scoring, 
i.e.. teachers) imist actually use scor Sng criteria to evaluate 

students' wrl^ng. Theses readers naist also be encouraged to comment 

r 

upon the ease (or difficulty) of scoring ^he sainples. 

Referring to the eighth criterl«?n, diagnostic Information can be ' 

provided only %dten the scoring -criteria require attention to the specific 

< » ■ ■ 

content features of tW writing task, rather than to the general writing 

performance or features of form alone. With only general performance assessed 
Knowl^ge gained of students* writing Is unprodiK;tIve for planning Instruc- 
tion; with only form features assessed, the form of performance may 
influence the assessment of content features, thus contaminating potential 
diagnostic Informatioft. ... 

Finally, the writing son^Ie prom^ ci:»td>ined with scoring criteria 
should yield evaluations that are stable across readers of the writing 
<;4tnple. Understand thdf no one is suggesting perfect agre^rmnt among 
raters; rather, evaluations should be reliable enough so they are unlikely 
to be idiosyncratic to a single reader (I.e., teacher). Teachers' Judg- 
ments based on long-terra contact with their own students* writing are 



bound to Mr more useful for classroom Instruction than any one-shot 
assessment; tait an advantage of using a survey across teachers Hes 
in inter- classroom activities, such as Informing next-term teachers 
of students' writing accomplishments. With each refinement of the TEPS 
writing saa^le as^ess-vnts, we ^thered data on Interrater reliability: 
we used classroom teachers as well as SWRL professionals to evaluate 
students' writing soaples. Resulting data were analyzed and thus recom- 
mended 'urther reflncwents. The remainder of this paper Is a description 
and n^rrat; - c* each major revision of the two TEPS writing sample 
assessments as they evolved through a series of field tests. 

Evolution of the Writing Sample Assessment, Form A 

The assessment Included the following note to the student, prompt, 
and scoring criteria: 



TMINtIKi Am WLOWEWT MIHEQUISIUS SUMEV 
CieiiSH ilTERACV i 

NOT! TO Tl«£ STl^CNT 

* - ■- ■ - ■■ 1 1 ■ ■ I II I — ■ 

The purp€n% of thii wrltifm ftMple is to find out what you 
hm^tm iMrneif ato*it writing. Write ymir name at the top of the lined 
pap^r Then read the dlrectims on ttw other side of this pape. Keep 
the dSrecilons In front of you to retnlnd yau of the things to include 
when you write. After you Tnive finished^ your teacher wHt collect 
your paper and this page. Tou mi U^ln. 
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OIUCCTiOIIS: StiKly ttm vmp of pmrt of m tchool gliding. 

LcruM ttm wmtm} thop and th« cftfetcrU. Vrltc 
direct loni (hat tall a sti^^t how to f^t 
froM tha Mtat %hop u> the cafataria. 



• Writa tNi diractfont la an orter that is eaiy to Follow. 
m im:ltMte all Important Mfonaatlon about whtch ursy to 

• include all in^tMt tnformtlOR abcnit wSat to Mtch for 
alofi9 thi i«nr« 

• Bo ftura to oaa axact words. 



WEST 




EAST 



y. — Lj' 
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MiTlim S^Li Scoring Critsri* 

Acceptaote Un#cceptatile 

CONTIHt: 

f . Sybils scorable Mfi^le. ^ ^. 

2. Indium rmportant Infortntt^pn 

•bMt dirtcllon of m^MMiit. ^ 

3. iMlttteft critical infomittion 

^KKit f««tiir«t along the way , 

k. VfiCM tflractlont In logical 

or4ar« 

5* U%9% pTMlao language. , , 

6* limltft the paragrai^ to ona 

Min li 



7- Uses correct grammar and 
ceit^late MantancM. 

iv^ 8* Capital I sa« «)d punctuatas 
dSj^ractly* 

9* SpalU correctly. 

10. Vrltaft taglbly with 

appropriate margins and 
irnlentation^ 



Before administering these materials to the first group of students. 
Me changed the scorify criteria to more accurately address the features 
of the particular writing task; that is, Umltlng the paragraph to one 
main idea (criterion six) was inappropriate for a writing sample that 
provides sequential directions. Vfe also removed criterion one (Mibmits 
scorable sample.) from the COHTEMT category, because scorability includes 
features of both content and form. Then, criterion ten (Writes Icqibly 
with appropriate margins and indentation.) was eliminated. Its function 
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A 



as a criterion for assassfng legible writing inas already claimed by 
criterion <wc (Sulmlts scorable sai^le.); furttemore, assessing margin 
formation v#as considered too subjective for easy or reliable scoring. The 
remaining utility of t^e criterion (asrsesslng Indentation) was simply not 
great enough for the time spent evaluating this feature in students' writing. 

Upon the recownendatlon of a SWRL model for evaluating writing samples 
(Humes, 1980>, h third rating was added to the scoring system: "Good-" 
So as we headed for the first field test, criteria for scoring the samples 
looked like this: 



MITIM U0^l Scarfnft Criteria 



I. Sytattft tcor^U taaepl*. 



CONTINT: 



2. IncludM Iflportwit 
4lr«ctlofi of mmmmnt. 



}. liH:lutfM critiMi 
fMti^M •lofifl the 



ik. I^lt«i directions in 



6, tises correct grmmr 
•M u>ii|>lot« MfitttimBS. 

pmctvmxm% carrot 1y. 
I. S^lit corrocttf* 
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The last step prior to aiialnfsterfng the assessnent was to expand 
the (tescriptlon of each criterion to include characteristics o'. each 
rating (Good, Acceptable. Unacceptable) for each criterion. The following 
directions^ accomplished this, task: 



_ _ m tfitatf to a«m tilt «rUI««.««9l« statoiu 

NMt mclMr* will only mnI to rM4 tho fvKloiiim « ^ 

tlM boforo tHof WMffbor criurtd fw' oacb mrfwmm^ rotff^. A 
•iiiot* ic^>i»« I* ^ovIM of tor tho ^itfollfiM. 

icartito fittid. cut tlM^ llil 
121. ItAltm ocofobio Miplo. 

Oootf: {TMt rotlf^ li not «»od for tttH crttorlonj 

Accoot«l»to: Mm bsSi of tuo folloMii^: 

«. tott^to wrftino 

wr'tlng ratovofit O dlroctlof^ In pronpt 

O. IliOfflMo IfTltSc^ 

b. «rUliif flot rolovont to dirttctionft In oroopt 

c, no rMpORM 

COOTfNT: 

la. lactirfw loporfo<»t lo^wi^ tloii iboot 4tnw:tloft of oo<o«if*t. 

locl^^ ot loost four of ttw fbttowioo dlroctloirtl 
lo^vnot*: 

a. (frw) oacol 



vlHO): oot ^ ftraloht otiopd 
fPvHollio twi > SI north 



Accopt^io: 



i. rl^ or oo»t 
c. loft o7* flortfc 
rt. loft cur wMt 

rlflUt gr MNth ^ 

tnclii^ jH of ttw iHrfo mjor rflrocMonol %m^pmM% 

a. (ot ftr»t *^ In liolhw)^ rl«ht mc 
l». loft gr oertte 
c. loft gr MMt 

Unocc^Uiblos M» opt ifictiiii £U of tho tfirpo oojor dlrocclonol 
•ogmtt Oft follooft: 

0. <ot-ftrwt **r*^ **on*<oy): rl^t or ooti 

1. loft |K flortk 

l#f t 9f Mftt 



^btrecttons refer to criteria numbered 121-128. These three-dlpft 

nimitiers correspond to the Ansvicr Sheet where ratings were recorded. In 
fact, 121 refers to criterion one, 122 to criterion two, and so forth. 
Note elso that the designation of the Form %i»s changed from to "A" 
to conform to the designations of the TEPS Pre-Algebra {Forms A and B) , 
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$11. twclw<i critical lnfoftmtlqw <l>»»t mtm^ Um mf ^ 



a, «ml Atop 



to. 

J* 



mHwU off lot 



•%«tl» ^tl»«y*> CM Ni ccmrM «Br» ttei mca If w» 
tallMy*' find *«)itfw« ^ Mtmf from ^ tmrot offlwi") 
AccvpMtoItt; HicIvAm tiir»» ^ fMl^rM Cm» tUt of fwtvm 

tlnou^tobUs liielodto* S ^ S5i <^ S£ fo«t«rw (w Hit of 
fMtum wi4ir oEmI. 

fa«. lirlw dlroctt« i« m taoicol oMof, 

All in«»f«otloR ift fftvwA <ii ordor of 



Ifi^nwtlOR it 9lm mtly In ord»r of oovomit 
(OM tMO difOctioool — to inU ond ^Mtitroi or« 

KlonMl OOt of of OOMMfll). 



HTitton MOoM mMio it diff Soilt Hr %iw 
itot to <bll«tf ^ 4lroctlM. 



121. mo> PTociM 



Prmtnm mf^ or* ittod In mt or oil cmrft to 
liitcflbo tflroctlom, fm* OM^to'^ 

"hoilMir** 'w** •>1oeo" 



fi o otr ot 
difoetkMs 
• foo 



•ro wotf la oofti cnm dotcrltoo 
(Soo OMioplm 



I oo y fc o in 



tniacctptoMtt «brtf» MMd oro to fftiMf^t shot o ooif t lodoot 

find It dIffleoU to foUoM tfco dimttm. 



m. oorfoct o r— Mr ood Cflooloto oootoacw. 



fiOOd: 



Hm of tlw follpwlnf: 

0. f«M Of flo giMwt H col orrort 

m tooooplo^ tonuocot {fr tpoi ntt. r«»-fift«} 



Mcoplototo: Mm ooy 291^ ^ folloitSiifi coototMtfont: 



f ow or no gftoMtlcol ormrt gjd fm loowtott 
Mmtoficot 

iM ffOMiotlcol orrort no loM^loto 
ioiit«i»« 

mo frooMiti£ol orrort £^ f«M IncGo^lott 

MNItOOfiOt 



Unsccimtoftlt: MM JfcJ^ of t*io Wlo»-l««^ 



mmy frtaMtlcol error* 
mm tA^Mploui tmtwicoi 
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15 



taotf: 9m fmp^m M»ltAtlmlcm end pmtiMitioii •rfPft 



•il^liiff rw* III tlmngH Itt or tlw mm* tum with l»t ffcrowi^ IIS 



111. SvfeiilU iCMMl Mlipl*. 

12}: Hmiwm iM*ltlMl lnfiHimlM^ iloirt «MttifW mim^ tHa wiy* 

ISS. prmlBm l imwg i> 

126. «0rf»ct iH«M^ CBWpltti* 

127. CapiullMi pimm^n mrmtly. 
I2t. IMIU cMN'fvstly. 
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Participants In tha field test for TEPS, English Literacy. Form A 
writing Sai^fe Mere 18 seventh-grade students (13 boys, 5 girls) who 
attended remedial English classes In an urban Junior high school, their 
four English teachers,* and two SVAL staff. Students were given tNe 
prompt and the note shown earlier. They wrote their responses on lined 
Mhite paper, some using pencil, but most, pen. All students had Spanish 
sumaoies, but their proficiency In spoken English could not be determined 
from these data. The fesults obtained froia the field test Included students' 
performances, reliability of readers' Ueachers and SWRL staff) ratings and 
their coranentary (reerfers* comnents for this and subsequent field tests are 
included In a SMRl report. Development of the TEPS Writing Sample Assessment). 
Students' performance by criterion and estimates of Interrater reliability 
follow. 
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Sumnary of Students' Performance By Criterion and Rater** 













Rating 










Criterion 




Good (A) 




Ac<:eptable(B) 


Unacceptable (C) 




Rift* R2 


Si 


R* 


Si 


SI 




SL 


R2 


Si Si 


1, 


2 




0 


0 


111 1^ 


18 


15 


0 


0 


0 T 


» 

2. 


6 


h 


9 


10 




. 3 


1 


5 


k 


6 7 


3. 


9 


7 


10 


6 


k 7 


7 


k 


3 




1 6 


k. 


12 


16 


8 


11 


1 0 


6 


k 


3 


2 


2 3 


5. 


7 


9 


18 


15 


7 8 


0 


1 


2 


1 


0 2 


6. 


0 


0 


0 


0 


7 10 


0 


5 


9 


8 


18 \} 


7. 


2 


1 


0 


1 


7 8 


0 


' 5 


7 


9 


18 12 


8. 


k 


5 


7 


9 


8 10 


7 


7 


k 


3 


k 2 



^rour teach ers dK-Jded students' papers, so each pair f 

half the papers. -For con^tatlons. the four teachers *«re treated as two 

'"^^'^iaters 1 «ird 2 are junior high school (seventh-grade) English teachers,^ 

'^'JI^Ritr'l'ciutSrran^^^ two students; hence the total number of. 
ratings for ^f^ter 1 Is 1*. 
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An examination of the data abov« shows a natural clustering of teachers' 
ratings ami SWRL staff ratings. All In all, the teachers' tended to rate 
mora performances good or acceptable than the SWRL staff members; however, 
upon closer study, th^ data show just the Opposite for criterion five 
(Uses precise -language.). This ijecullarlty prompted a rating sequel that 
involv^ two other SWW. readers and the same student responses. The 
.results of this siaalt-scale rating activity are jdescribed later. Except 
for criterion two (includes la^tant Information about direction of 
novement.) and possibly criterion three (includes tritlcal Information 
abou^ features along the way.), these coefficients mre.dlscmiragl g. 

V 

Summary of Estimated Interrater ReUablllty by Criterion 
Bett^en Pairs of Four Bsters 
(Pear^Mi CorrelatfcMi Coefficients) 









Raters* • 






Criterion 


1 2 


} ^ } 


1 & * 


2 & 3 . 


2 & * 


3 & * 


#1. 


.1336 


** . 


.(K)00. 


■nil 


-.1195 






.67*1 


.6362 


.772* 


, .7699 




.3081 


3. 


.51*3 


.6192 


' .<t883 

« 


.5**7 


.637* 


.63*0 


k. 


.2936 . 


.5*30 


- .0136 


.^768 


-^2579 


.2553 


5. 


.33*3 


** 


.0513 




.6029 


** 


6. 


.*535 


** 


-.081* 




.3051 


** 


7. 


.2<(*6 




.0377 


i 


.*a9i 


Aft 


8. 


.3280 


.3**^0 


-0397 


.8*6* 


.7755 


.7330 



if 



^Raters I and 2 are junior high school (seventh-yrade) English teache 
Raters 3 and * arc SWRL staff. ' - ^ 

A»Coeff Icient could not be coBipUted because variance was zero. 
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So'm« located at results for /Individual students'. A^d there. It hft us. 
If "Good" ratings were collapsed with "Acceptable" ones, agreement between 
pairs of raters looked es If It would skyrocket. thtsoob5ervetl0n was 
set asl^ tewporarlly while the Fdna B assessaient was fleW tested; It 
too yielded such results. During this tlsie, readers' coiraiants about the 
Form A assessment were reviewed. 

Based on student performance, estimated lnt.errater reliability, 
and r^ers' commts. the note to the student, tte prompt (Including the 
diagrwn), and the OIRECTIOHS FOR SCORIMa THE WRITING SAMPLE - A* 
(now IncltMied within a draft of the administration manual, wider sections 
"Administration: Writing Sample" and "Directions for Describing the 
Writing Sample -.Form A Guide amd Form A Summary of Descriptions") were 
revised as follows (the Note to the ,Stu«tent was now designated as Form A 

♦ 

rather than Form 1): 



INSLISN LIltSACT A . 

imiTiiis s*»^ ' ^ ' 

TO Tttf ikmn 

The pwrpoM ef thit n^rUing sMpta is to find out what you * 
Hm* iMhiMi tkoAt Mrltinf. mitm vwr nam at th* of ttw Hnrt 
p^r. Then rMd tihm tfirccttpnt on tlM other tite of this pops. K«ip 
th« dirMtiofa Iti front «f you to mli^ you ef th« things to Inclo^ 
MtMn yoii urttm. Aftmr yM Imv flntshi^, ymir tMchor mIH collNtt ^ 
your i»9or ml this PHI». Vou mmf boglA' 



^Directions refer to criteria numbered 111-118. These three-dlgU 
numbers correspond to the Answer Sheet, where ratings were recorded. In 
- faqt. III refers to criterion one, 112 to criterion two, and so forth. 
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DIWECTiaiS; St«M^ tlm m|> of pmrt of a school tatl^liii* 

a ptragra^ that tot it o Mitf tturtent tow tOr^t 
from tto ««£tot ^op M tiNi col^torla. 



o> VriM the dinl^tloRS lo on orAif that It eoiy to follow; 

• Ineliidv oil IfiportMt Sofofootloo «^oiif i^i^ Moy to oo. 

# iiicl«ftf« off llR^rtoiit lofornotion idmit-irfMit to Mt^ for 
•looQ tha mKf4^ 

o Be loro to m« 

moct tuirtffi 

*^ood ^fMBor on4 con^lote MOtoncoi « 
corroet ^^iN^toation and spellif^ 

W»TH 



mia 



1.^ 



M»ITpfttUH 



i laniu's ' 



'I 



EAST 



__1 



cm 
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« 

offsrt.M oppprtmity to 4Mcrlki charKUrUtlcs of on ln4Uldo»r» 
tfrittoo rai^omo to^o f^Wt* Clioroetorltll^t* dMcrilHN! iMliMla cotitent 
(liSooft roproM^tod) ofid f^r^ (lootuofa uft«^ mnd mchonlcft). 

^ o tM WrUliit Soo9*« Is not lotoikM «t> to o tlwfA octUtty* 

Hftlilii rooMHiAlo IMti, Mcli tMMlnw ohwN te «lvon oomgh 
tloi (otout IS*25 fliimil^t) to cooploto tho itoo, 

o Mmmr Uioots oM leoklots fthmiltf m te oi^t lobto to 

OMploooft d^lBt tlM mMoiotrotfon of thiVri^os So^lo* 



OUtritoM to Oftcti ooMlooo tho tio^lo ohett mrkmi VfliTif^ 
SMKE (Rmi H or t) « ohMt of I ItmA pmmr. ^ 

o HOM omAoms foo4 tiio loformtion in tNi boii bolofe* th« wor#t 
MTI TO TM S1«^* EAninOM ihouid «&i oblo to rnd lodofMn- 
4mtly oAd folloi* tiio diroctlom for ^itiiif m^Io. M 
ovro oach mmmlmm i«rltt«n hor or hlo Mae oo tho llmsd 
|M^r, 

• ^ 

o AtAr oaotttfiooft flolsh wrltiiqi, colloct tte single ^ihoot of 
diroctlooo fm* t^ WITTW^ »m.l o«^ ttio sctMl wrtting %m^\m. 

DtroctlQt^i for OoscVlMoa tto ^Itltia Sawglo {¥wm ^'mvi %) 

ToocWt ora Mtuird to dncrlte tho wltlng onplo lAlch iti»N^nif 
OMplotod OA tho liiml fropor. Cuidolff^M for doscrlblng ths %irltls^ 
ftoe^lo oro iHROtMtod bolm. Vqo shwild wio sitrfwts* M^wr shwts 
and fin In tho ftPQgoo oncloiod In thtji^ yf_!^„g^| ^l 

. nm foldot koldW (Nmi* A 4^ S) doscrlbo otudent porfonmtco fpr 
ioyorol eii»roctorUti^« Rsfor w tlMi j^ldoo to fill Ip tl» tobbU 
idioM doteripllon com olteoot M tNi otodoi^t'o porforwMO for ooch 
ehsroctorUtU* 'A 0M*p»9ir luwory oC OMCrlptloos foMoM iha Oylda 
for ooch form. TNi Smmrf »fi ^luifilify tlio toftfc of d»»cribtn« 
cKoroctariftlu« 

Form A Soi^. ^p^too m^diorcd III through 118 ore Msed to doscrlbe 
cHarKtorlttUi' of writing tl^l«« for form A, Us« the d««crIptlon» 
hclQw to fill In tho bi^Uft on tHo tt4id«ntVs onsi^er %f^x. fM*, In 
only one btd^l* for mmch nombor. O o not fill In any hiibfale* if tte 
>4wpU l» ony of tho follo>iino s 

- II legible 

* - Irrolevont to dlrectl<ms given 

- not written 
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^^^^^ 



(A) 



CO 



to) 



n 11 



llrifM ateiif Vi'lm ftbMt Writm aboyt "^Ir I t«f atKNit 



If hr^itf^itln^ U imiCh larger or snaUcr ttwn avera^. 
J^d M^lt fin KP>rtfln9 to i4«t It wow Id be If 
timfefrlting w&tq of iivorogo , 



m. ifteludw Iqportoilt 
3Trocllow fT 



11%; 



tnclwloft all df tha throo m}w 
dimctlocMill M^wits: 

- rf^t or w«t (at f{r»t 
liTlMilbMy) 

- loft or north 

* - llfft W MSt. 

In€lu46i throe or woro of ttw 
fol tmiiig fMturfti: 

Mtot fthc^ hall or IwtlMf^ 

cookififi ro^m ort rooa 

^ llbrory 

whool ttt>ro luirM'ft office 

oodltorlym feMret office 

coitetorio door or 4oormf 

tMTI- *'hetl" or **hollwo/' con bo coimled woro thon^onoa If «oro 

thM ooo helliaef U wMtloned Cfor owaplet ' 
hoilW*' ^McrMt tl|e HollMoy frtm the ,^rol office 



incleioi cl-ltlco l 
itiforwitioo ^oMt 
l^toroe elCNHi 



(A) 
Yoft 



(S) 



In loplcel order . 



Afl or fiiwit' InforwatlcN* \% given 
Ifi order of noveoent. 



(B) 
Nti 
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IIS. tfw grwiw * «M» »r«clM wordi to <N»criNt 

" dt wctfonm, for 



(A) ^ (■) 



FQiW: 



m. cof r^t anmmr 'Tlw .total iMi«to»f of sraanatftal 

. errors l» • 

. • , " • ' ^ : 

. (A) (9) (t) (D) (t) 

mm 2 J. ^ 5 or 
* ' - or 1 " more . 

•HQlIEi Count on* •rror Mch tin* a Mgrniiit of • f*m~oa sentence 
•heuN bem a M^MNi»ta «nit«iic«. ' 

C«)ttailaa» aad ( ' Tha total nunber cap! tali nation 

wctuatoi corractfr. and punctuation error* la ' • ' 



117. 



(A) (t) (O (») (t) , 

nona 2 3 * 5 or 
or I WW'* 



•WTI: 6o not eowit ai ffroNt tha |»lacas> Miwra a rua-on aantanca 
ahttuT? ha^ bawt sagMitad (and tlwa capital Ind).- 

118, Saaili c«r'ractl». THa total nuabar of apalllng;^ 

^ vrrora la .* 

OD (B) CO .{e) (f) 



AOM 2 } k S or 

or 1 ^ vtjofw 



*Wm: Hor« ifmn fom \nmtmfcm of Aisi|Killinfi the %mm %<ord the 
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ikmmX fiMt to 



tdtf W » tiio JlroctloAAt to holp you fill In the 



in. 


Soiiplo sin 


(A) 


(B) 


(C) 


(0) 




* 




full 


Vk 


1/2 


1/* or 
less 






Qlroctton of mmmtnX 


(three so9«»nts) 










¥es 


No 








113. 


Critical futiiroft 


(Ai (B) 

(three feetures ->-) 










Yes 


Mo 










• 

LoBlcul order 


. (AJ 
Yes 


{•) 
Ho 










NucIm Imtii^ 


(A) 
Yes 


(B) 
Mb 


^ 






116. 


Errors in grewMr 

end Incaiplete sentoims 


(A) 
0-1 


(B) 
2 


(C) 
3 


(0) 
k 


(E) 
S* 


117- 


Crrori In ca«>l tellMtloo 
end fiimctiietlon 


(A) 

0-1 - 


(B) 

2 


(C) 
3 


(B) 
\k 


<£) 


118. 


Errors in lulling 


<A^ 


(8) 


(C) 


(D) 


(O 






0-1 


2 


3 


<i 


5* 



* notice that the prou^t re)w spelled out the requirement for exact words, 
good granwar and con^lete sentences, and correct punctuation and spall Ing. 
Notice too that symbols for two exterior (and extraneous) doors *«re 
removed. A minor adjustment was made In a line that represents a wall 
'Of the principal's off Ice— previously that line ran through the lettt. 
5 in principal's . Another minor change *«s that the form of the TEPS 
mentioned In the heading of the note to the student was changed from 1 
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to A to correspcmd to the Pre-Algebra con^ioncpt. The revisions made upon 
the readers' directions, ho«^ver, Mere major. 

It occurred to us- that readers may have beer scoring the writing 
'sainples according to habit, rather than attending to descriptions asso- 
ciated with individual criteria; choosing ^ong ratings called good , 
acceptable , and unacceptable , and recording these as A, B, and C seemed 
as If they could reinforce custoinary paper-grading practices. Since the 
Intent of the TEPS was to describe student accompi Ishflsenrs and not so 
much to rate or grade them, a .olutlon we decided to try was a descriptive 
rating system. In addition, individual criteria were revised. The high- 
lights of these revisions were as follows: readers were directed to set 
aside papers that^re unscorable (according to characteristics given) 
rather than darken a bubble for a criterI*on; the size of the writing 
sample was to be estimated (to eliminate the vague, judgraenial descrip- 
tions for the form criteria, e.g., few , some , many , and several, by 
directing readers to count errors for these criteria— school systems 
could set their own error standards based on size of sample); and content 
criteria could be rated only as present in the 5.wq>le or not present ( yes 
or no). Also, each form criteria was. follo*««d by an annotation that gave 
raters additional guidance (especially In dealing with run-on sentences). 
Two SWRL staff used the revised directions to describe the san» 

m 

writing samples. Without a group of student papers that %#ere In response 
to the new prc«npt, of course, fue^ data would be limited to new rating 
directions and new raters. A similar wording change for the Form B 
prompt, however, *^ field tested and did improve the results. 
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The following suBwary of students* performance and interrater 
reliability estimates reflects revised crlteiia and ratings from two 
SWRL staff who hadn't before seen or heard about (I was assured) any of 
the materials Involved In the writing sample project: 
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Cri terlon 



Summary of Students' Performance 
By Criterion and Rater (Revision) 

Rat I ng* 



R1 R2 



B 



Rl R2 



R1 R2 



Rl R2 



Rl R2 



1. 


0 


0 0 


0 


7 7 11 n 




2. 


11 


13 7 


5 








16 


15 2 


3 


> 






13 


15 5 


3 






5. 


11 


\k 1, 


k 






6. 


1 


2 6 


5 


. i» 5 2 2 5 


k 


7. 


I 


1 3 


2 ' 


3 10 1 11 


13 


8. 


6 


3 ^ 


5 


2 3 0 1,6 


6 




Sumnary of Estimated Interrater Reliability 
By Criterion Between One Pair of Raters 
(Pearson "orrelatlon Coefficients) 








Cri terlon 




Raters 1 and 2 








1. 




1.0600 








2. 




.777^ 








3. 




.7906 








^. 




.7211 








5. 




.0311 








6. 




.637'* 








7. 




.630<» 








8. 




.7887 






-An absence 


of data under a rating indicates a rating not described 


for 


th^it criterion. 









criteMDn five- It looks like an error In marking the Answer Sheet. 
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Except for criterion five (Uses precUe language.), these data were 
considered acceptable. Though coRq>aring these coefficients directly 
with those resulting from the first version would be a faulty statistical 
procedure {after all, the criteria, raters, and directions changed), It 
isn't faulty at all to peek at a promising contrast, especially in the 
readers* agreement on ratings for the form criteria (6, 7» and 8). 

Remaining tc be dene was an overhaul of criterion five. Why did 
the raters differ so markedly In applying this criterion to the writing 
samples? As I saw it, the' only way to ansi^r the question was to ask 
tire raters — so I did. One SWRL rater focus'-U on the example given In 
the descr ration, i.e., "cooking rooirf' or "rooni" (r.ther than "place"); 
the otlwr rater focused on the seciwid exa^le given »n the description, 
i.e., "left" or "north" (rather than "that way") . The next step was to 
siee %tfhac wculd happen if we mnoved the examples from the description 
for criterion five, r»*rote the description, and then had tvto new raters 
r«ad the {Mipers (s«ne ones) and select a rating for this criterion only: 

Criterion Pescriptlon 

Uses precise language. Host terms used are exact. 

Vague language is avoided. 
(A) (B) 
Yes Mo 

The students' performance according to the tw) raters (SWRL staff who 
haitJ not yet been involved in any phase of the writing sample project) 
was as follows: 

Suiwnary of Students' Performance 
0.1 Criterion Five by Two Raters (2nd Revision) 

Rating 

Yes (A) Wo(B) 

12 9 6 9 

30 



The interrater re iability estlmati^ to be .7071 (Pearson correlation 
coefficient), which indicited a more consistent Interpretation of the 
criterion and its description. 

Review of comnents from readers involved In the revision led to a 
note for criterion seven that reads: "Do not count as errors the capital- 
ization or lack of capitalization of room naiaes (since much variation 
exists In the capitalization of room names on a building plan)." Three 
other notes were added to this criterion on the basis of recomrocndations 
for the Form B assessment. The description for criterion five was changed 
as noted above. 

Evo lution of the Writing Sample Assessment, Form B 

The TEPS. English Literacy, ^orm B writing sample assessment began 
with the following note to the student, prompt, and scoring criteria*: 



The purpM* of this Malting B«pl« It to find out «d»t yoti 
h«y« iMrnsd rtttit writlrsf. UrlM your mw at th« top of tta lined, 
fwpcr. Th(W» rwM tlw di ration* o« thm 0thmr %lim of thl» p«^. Kmp 
th» dirwctiont la front of fon to nwlnd you of tho thln^ to InclMto 

you mrit: ktimr yew htnm finlihwl, your tMdnr will col lost 
yottr pspar «^ tills 9^1. Iteti b«v bosin. 



TmimW AM IHPLOVHiliT WUfWISITES SimVEV 



WITJM6 JMHPU 



NOTE TO THC SnWENT 



to the Answer ineet, wncre raiins* • > — • — 

criterion one, 112 to criterion two, and so for 
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OlttCTlMSs Vmt wrk In ttm off Im of • lar^t •l«ctrqnlc» co«i>«ty- 
IWo liirti your coMpMy onten^ note p«dt from t9i« 
TttM Ffint Shop, fou rocoivod the shipimit of ooto podt 
tliU wmioo, but tfioy ore or in tod mI^ tho wroog tolo- 
ptoM miftor, Wrlto o bMitiMft lottor to tto Tltt» Print 
^^fmp. lm%\ ^hm tfmt fou oro rotomlni tlm noto ot^ md 
thot yoo Mot thm to print oom onos ot no wMUIomI mt 
tik votMr CMOon¥» 



o THo 0|Mmifi9 mtf cloftii^ porta of i^r lottor oro provldod bolow. 

Wrlto jMt thi tody of ^ lottor. 
o Toll idiy you ftro wittofl tiiit lottor« 
o Stoto «IM too oont tiio irtwp to do i*Mt ^ pnAlm* 
o lo Mro to i^A'oxoot ifo^^* 

o lo ftoro to iiM mHhk thmt oro ^^t^loto for o bMlnosf totter. 





























1 
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T«KhM« mrm ^b94 to tcttr* ttm *M>itlii9 Mopl* mHI^ tti^tt 
CMVlattd m tiw Itnttf MWr. 6iiltf«linM for gcorlng t(i» Mrftlns swl« 
•r« pmnt«d NlM. Itew rtwwld do »wriiifl dIractW ow ttw back of 

Hic« rating* *^ Mdi critsrien: aopd. accwtiibla. and aycctpfble. 
nil In mrtmi * for jogjl^. Bfer i^^t^Je. antf C for 

MMt tta^fB will only nMd to rMd tlw teorins gwldalliMS • f«w 
tlMM ktfora tlwy riMwhar the critaria for oadi parforaaitcc ratiiH). A 
slapla tcorlng aid Is provided after tha «tiida1inas. ' 



Scorlwa fiulda (111 thrpyqti 129) 
Ifl . Sabwtta acarabla aawla. 

Bood: (THift racii^ It not mdd for this eritarlon.) 

Acd^tabla: Mai both of tha fellowliHi: 

a. lagihia itrlting 

b. ««rltinf ralavmt to dlractlens in pronpt 
toaccaptabia: 'Nks any «m of tha following: . 

a. illaglbla writing " 

b. writing not ralavwtt to dlnretloni In prompt 

c. no raapenaa 

CMTENT: 

r 

112* Sf ft rwow for l<tt»r> 

liwIutfM wv ^or thraa of tha following: 



finod: 



ordariag mni racalving nota pads 
wrong tal^hoaa m«bar on nota pads 
ratofiiii^ nota pads 



Accaptabia: Includas any ana of tha following: 

a. wrai^ tal«*iareS wirtbar an nota pads 

b. rataming nota p«bi 

Unoccaptabia: inclMdM naithar of tiia following: 

a. wrong talaphona naafcar on nota pads 

b. rataming nota pads 
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It}. «fffm *t»lr^ »^ -y^^y- 

iMltttfn Imth of ttit follMlnf: 
ft, print Mn Ml* (r^l acTOW tty 



Jtecopti^lo: IkIoAm: ;irliit mm Mto jpod» (roplOMMiiu) 

UMCc^toblos &MS Mt Incltite: print.MM notm podi (mplKeMntt) 

*i 

IH- liM» OrOClM iMOMOi^ 

PtmIm mt^ or« tmd In Mit or all cmm ^ 
dMcrfbo foportont fMtttrM of ttin lottort for 
OMW pIo: 

'*Mto po«i'* vn. *^podft*' or "^^ii^'' or *Hit«tf^* 
««prlntttf kW* vi^ •Vlth^^r ••how" 
•'tolophoM ntfi^r** ••miflbnr*' or *VrUInf*' 
•^Itloml cost** vi. **co»t** 

Bmorol wrte om m^ mt c^m to ^ncrlNi 
Inporumt fMtom of ^ lottor, NoMtrer, pmciM 
wordft M¥^ bo tiMd In o fov cmm. (Sm MMptos 



iknoptofcio: 



l^copt^lo: li»r^ mod oro M 9nn»rol tHnt ttit i^ipiont of *«tlio 
tottor Mold not bo Alt » csittefstwrf Vm rMiton 
0Ml1iit« for OMipIo: 

*^$m ptiiiy Is m OMd to here It U*' 

ifS. tftw Imomoo oaoroorloto far o bMlnof lottor . 

Good: Hm both cf the fatioitlnp: 

o. poll to or nootral tono 

h. lopofMMl ttM (^ft Mt rofor to porMn irfho 
hMdIod midOr or Mlf) 

Accoptnhlo: Mto £o|h of thn follo^tlnf : 



a, pollto or MNitrol tOM 

hp pormol tOM (rofor« la» por»on %dio hntidlnd 
ordpr or Mlf) 



VnMcopti^lo: Mm t^Mto or nids toM. 
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ijiclii«M wir«Ut«4 MiitMt. 






All fntHMii rafsr to titMStnti teicrlkM in pranpt. . 










• 


NMt wwtmt^m do DM f^v^r m tttufttioniteftcrlbod 
"In profit. 




Htm C9rr*«t a 


trMoor and foi^loto oontoiicoi. 


Sood: 


MM both of tho folWin^: 

o* m m growMttcol orrort 

m iwmfimtm amtmooji (frofmntft* nm^») 






flM any ooo of %tm following cooblnatlcms: 

i . 

0, or no froMotlcol orrort md fow lonoploto 
tontoocn 

6. looo troMiiCicot orror* £^no tm^o^ploto , 

c. ton groHwticol orrorft joid fow Inconplotc 
MfttoncM ^ 




•* 






Nn both of ttio followFiig: 

p. mmvf groNMitiMl orror» 
b. flony tnoooploto ftmt«ico» 


IM. 








Wm fm pr m e^ltoltmtcm Md ponctuotlon errors. 






ffOM coital intioii pmttumtitm mrror%. 






iteft*OMif c^ltolisnion ond pitfietiiotion orrors 


119. 








Hm foM or no oUipollod iriord*. 






Mm MVdrol difforont* nltftpolli^ wrdi. 






wmnf difforoot* »fifpo11od Mordi. 


tHM on* IwtMM of aitspanine ttm tmm mrd •houU bm 
ovsliittCtMi OS oRc R? sepal 1«l m^^. 
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Icerlna Aid (111 thrBMh 129) 




ttm» Vim •tiMhmc** mnmr 
alffRtnS rpM 121 tiiroiiah 119 m 
tel«M: 


»itaat undar titl* pf of ttw Manual, 
tha anSMar tliaat mith 111 tUrM^ 129 




(A) fiood 

(B) kBtmulkim. 

(C) liM^wtil>l» 


StdMi 


€« M»ri^la MMpIo* 




l|2. States rwrnwem for lauer. 




111. SMtM 4Mini4 CKSTM of actian. 


114. ttm pnclM l«np*Ml«' 




IIS. tiMt 


laagiiata wnN^lata for a btnlnat* latter. 


IM. Exclute tnralatad tantaacas. 


117. 0»a» cerrKt trmmmr «Ml conpl'K* tantanca*. 


m. C^ttaHiM and iMmctuatat rarractly. 


Its. S<^llt cwrractly. 





By the time these materials *^re field tested for the first time, 
they had already underQOfte many revisions. Review sessions with several 
SWRL staff resulted In the decision to eliminate son» criteria frosn early 
drafts (under COHTENT) , "Identifies self and cc^fjany." "Limits paragraph 
to one main lr!ea," and (under FORM) "Writes legibly with appropriate 
margins and Indentations," and to combine criteria (under CONTENT), 
*'States purpose of letter," and "States reason for dissatisfaction" 
(which later became "States reason for letter"). The two content cri- 
teria were eliminated because they were considered unnecessary or 
inappropriate for the particular writing task. The form criterion was 
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wIthdraMn for the same remns as It was for Fofw A: the first criterion 
alraatfy t^ care of leglblltty, margin fbmatlon could not be scored 
easily nor reliably, and Indentation alone had Halted value for assess- 
raent. The criterion foraed froa two ««as simply a move toward economy: 
reviewers didn't see nuch difference between them, given the nature of 
the i^ritlnf task. 

Participants In the first field test for the TEPS, English Literacy, 
Fona B Writing SasipU were.%0 tenth-grade students (22 boys, 1 8 girls) 
Mho attemted reinedial EnfUsh classes In an urban high school, their 
four English teachers,* and two SWRL staff. Students were given the 
prod^t and the note shown earlier. Their responses were written in pen 
or pencil on lined paper. Like the students who participated In the 
field test for Fona A, these youths all had Spanish surnames. Following 
are data Indicating stuftents* performance 'jy criterion and reliability 
of readers* ratings (teachers and SWRt staff). In looking at the 
data, note that raters 1 and 2 (the teachers) assigned a rating of 
Good for criterion one to 19 and II students, respectively. The 
problem Is that Good Is not a rating option described In the scoring 
guide for criterion one. We can't tell whether the teachers lnt«ided to 
select Acceptable (Instead of Good) and Unacceptable (Instead of Acceptable), 
but followed unclear directions Incorrectly, merely disregarded the scoring 
guidelines and fell Into the customery practice of judging students* papers 
6o(Hi, Acceriable, or Unacceptable, or came to these ratings via some other 
route. If only one rater was In error, we might have simply acknowledged 
it as a singular occurrence. 8ut two raters doing It warranted action. 

*The ^our teachers divided students* papers so that each pair of 

teachers read half the papers. For computations, the four teachers were 
treated as two raters. 
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Summary of StiHtent-s* Perfonjance 
By Criterion and lUster 

toting 



Criterion 








Acceptable (B) 


Unacccpt^le(C) 




R1 


«r 




nk 






B4 




R2 


£1 R4 


1 




M 


0 


0 


20 


28 


40 


40 


1 


0 


0 0 


7 




26 


37 




17 


13 


3 


6 


0 


0 


0 0 




2% 


2% 


34 


2k 


15 


13 


6 


k 


1 


2 


0 12 


k. 


2<» 


2k 


16 


29 


11 


9 


24 


n 


5 


6 


0 0 




2k 


Zk 


2 


0 


10 


10 


33 


30 


6 


5 


5 10 




27 


29 


39 


37 


13 


9 


1 


3 


0 


1 


0 0 


7. 


10 


13 


% 


19 


19 


17 


25 


15 


n 


9 


11 6 




12 


14 


12 


30 


16 


\k 


20 


7 


12 


11 


8 2 




IS 


13 


16 


30 


13 


15 


23 


7 


12 


10 


1 0 



And the substantial dlffen»ices between raters on performeince Judgments 
for nany criteria, not only bet%*cen the terchers and SVHIL staff » but also 
between the two SWRL staff (the tochers tended to assign a similar nund>er 
of tH« s»ne rating for individual criteria), called for a closer look at 
the data. Miat follows Is a simmary of estimated reliability between 
pairs of r«ic"ers. 

^Raters 1 and 2 are high school (timth-grade) English teachers. 
Raters 3 and 4 are SWL staff. 

^fteter 2 CNsltted ratings for one stiMtent; hence the total nunH>er 
of ratings for rater 2 Is 39. 

^Rater 4 omltttKil a rating on criterion elf^t for <me student; hence 
the total mKi^r of ratings foi rater k for this crlterlm Is 39. 
d 

Rater 2 omitted a rating on criterion nine for two students; hence 
the total ntM^r of ratings for rater 2 for this criterion is 38. 

Rater k omitted. a rating on criterion nine for three students; 
hence the total nio^r of ratings for rater k for this criterion Is 37. 
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SuiifiaryMof Esttneted interrater Kj^liabUlty 
0y Crltarlon fietween Pairs of Four Raters 
(Pearson Correlotion Coefficients) 



Crltorlcm 


i ft 2 




Raters* 
1 ft k 


2JJL 


2 ft <i 


3 s k 


1. 


.2001 


** 










2." 


-.2S80 V 


.3312 


.3*70 


.0822 


.05*2 


.6778 


3. 


-.2717 


-.0710 


.1075 


.W5 


.2303 


.2956 




-.1821 


-.0431^. 


.3347 


-.0269 


-.0303 


.0*57 


5. 


-.1590 


.1105 


.5072 


.3859 


.15*5 


.3156 


6. 


-.2879 


-.nil 


.0051 


-.0908 


.*nii 


-.0*56 


7. 


.15*6 


.5192 


.4H7 


.1578 


.2209 


.5*83 


8. 


>.0iil0 


.2766 


.22I19 


.4509 


.13*7 


.501* 


9. 


.I6I15 


-.0072 


.32ti8 


.3116 


.4878 


.3605 
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The data above were truly discourpging; the degree of agreement between 
pairs of readers was so low It mos difficult to believe that both readers 
looked at the same papers end same scoring guide! (nes. Since \t was a 
fairly simple iMtter to recompute these f Ijpires, I.e., arrangements for 
field testing Mtiran*t Involved, we treated the Ciood and Acceptable ratings 
as merely Acceptable, and tlw Unacceptable ones as they stood— we collapsed 
the rating options from three to two. Mote the following results. (But 
understand that the results represent a paper manipulation and not actual 
ratings.) Immediately, you can see that these data were less Informative 
In general than the data just above. I.e., with Good, Acceptable, and 
Unacceptable ratings; fewer coefficients could be computed fcr^tlw^ 
collapsed ratings, 

AReters 1 and '2 are high school (tentii-grade) English teachers. Raters 

3 and 4 are SWRL staff. - , 

**Coeff Iclent could not be computed because variance was zero, 
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Sunnpry of 'Esftl mated interrater ilen:;bM!ty. 
By Crlwrloir^tween Pairs of Four Raters- 
(Pearson Correlation Coefficients) 

Collapsed Ratings 



Ratei^s^ 



Criterion 


1 & 2 


1 & 3 


1 & k 


2 & 3- 


2 & ^ 


3 & 4 






*ft 










2. 


ft* 


** 




r 


** 




3. 


-.0377 


«ft 




** 


.0969 






-.1635 


ft« 




Aft 


^* 






-.1635 


.0529 




.5*12 


.1261 


e3055 . 


6. 


• ** 


** 






** 




7. 


.062i| 


.3730 


.2117 


.3329 


.2725 


• 3685 


8. 


.199* 


.2182 


.0969 


.**92 


• 2751 


.*9Z1'' 


9. 


.1083* 


-.10*8 




-.0982 


ftfi 





The only exit from, the fwedlcaaent, as we saw It. was to revise the 
mater iaU and then field-test them with are>ther group of students. For 
direction toward revision, readers' conwents were reviewed. * 

Performance data, reliability data, and readers' com^nts led to 
the revised directions, writing prompt, and scoring guidelines*** below 
(the note to the student was retained as written): 

*Raters 1 and 2 are high school (tenth-grade) English teachers. Raters 

3, and * are SWRL staff. « 

^ **Coefflclw»t could not bte computed because variance was zero. 

***Culdellnes refer to criteria numbered 111-^19. These three-digit 
numbers correspond to the Answer Shett, where ratings were recorded . 
In fact. Ill refers to criterion one, 112 to criterion two, and so forth. 
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mUECTKHK: Vov work In ttw office of a electronic* comity* 

Two Mi^i i^, ifour comity ordered towe supplies fro® 
Che TitM Office Supply Con^y. You rectl^d the 
fthlpwit thU coming, tet It wet not whet your co^ny 
hed ordered* 1^1 te e iHiilrait letter to the Titus 
Office S«wly Conpmy. Tell thm ftd^t Mt with 
the fthipniNit. Tell then «diat you went thew to do 
rteut It. 



The open log «ftd doling parti of your letter ere provided 

below* Vrlte just the body of the letter. 

Tell enKtly whet wet wrong w'th the shipment, 

Stete enectly %^t you went the Titus Office Supply Co^ny 

to do ehout the shlpMot. 

Be sure to UM 

oxact i«ords 

words ttwt ere ^^opn^rlate for « business letter 
good gremer emi coi^lete sentences 
correct pimctuetlon ihmI spelling^ 



IttI lii«itttrl«t 



ftl»f fir or ^ 






L 
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rofm 1 M4m . Spmcm% nve^rm^ 111 thrmtgh 119 mrm to <te»crlte 
clNirwMrliiti» of tm wrttlim tMpU for form 4Im( th« descrlptlont 
tolow to fill la elm on tiw »t«i^t** msMr ttiMt. FHt in 

only om buMit* for mmh fumter, r ^ot flit In ny bullet If tho 
Mwplg l» »HY of th> follwltifl : 

• IrrattMHit te rflrwctlen* given 
- net MTlttan 



111. 



_iiit 



(A) 



(!) 



(0 



(0) 



Mwplc {bnmi WrItM abmit tfrfta* altoMt Writes about WritM about 

1/y K II" B full pas* J^* 

tbiAt of or «»r«. or less, 

r).*- 



•tlOTI: ff hKMlHrltiiis U wuch lar«»r or wwllar ttwn •v«r*«B, 
J(Mt«* MHipl* ftta M:cortfln« to trfwt it Mould be if 
hondMrttins Moro of ovarage its*. 



CfMTENT: 

112. $teta» pr^lw». 



HaatioM a siwclflc error or 
pr^lM Mitb the shtpmnc. 



(A) 
Yea 



(•) 



111. State* dw»i*'*f 

cour— of actTon. 



Hentlona a apeclflc Mtien that 
tba office aivply ca)«>MV »beuld 
t^e cancarnii^ the sbipnent. 



<A) 
Yea 



(B) 
No 



11*. tftaa precia* 



Ui«« prociia Morda to Macribe 
I^WtMtt point*, for aMMipte: 

"pmnt," or '^ka" or "chain" 
(rather than '^hipiwnt" or "thinfa") 

•WoAf color" or "2 4o»mn" 
(rather than "mo/^' or "not 
enowffh") 



(A) 
Tea 



(1} 
No 
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lis. 




Uses polite lerrguefs. 






(A) W 
^% to 


116. 




All or mit sentences refer to 
sttMtloA described in DIRECTIONS. 

(A) (B) 


FORM: 






II/- 


UiM COffKt flffftMr 


Tte fftel member of gresmitUel 
mrrw% md Incomplete sentence^ 
errors is 

(A) {%) (C) (0) (E) 

none 2 3 ^ 5 or 

or i Mre 




^HiOTE: CcHmt vm •fair MCh tins a i^Rmt of a rim**Of) i«nten€« 


ns. 


Cwlt»tlsM mi 


nie totet nmber of cepltellxetlon 
end BtffKtyetlon errors Is > 

(A) (B) <C) (D) (E) 

none 2 3 4 5 or 
nf t wore 




^ilOTC: 09 Apt OHint M arrofft ihi i»l«CM ««fcer« » run*fm 

MfitiMe should m tmmn tegntnud {and thus c^Ualixvd}. 


119. 


SMllft corrvctty* 


The totel nw»ber of spelling 
errors Is ** 

(A) W it) W (C) 

none 2 3 ^ 5 or 
or 1 MDre 




^mmt: Hot* tim OM ioftUmctt of niftftp^llifig the Mm word thm 
%mm My »Kmld bm AMlwatad m Just one misspelled word. 
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Mmlnl»trmt\Qnt Writtwa SaagU 

Tfm tfrlti«fi Smi^I* It tlw part Vf tlw fnt><«<> LiMracy Sttnnay that 
off»r» an «p|MM-t«iilty t» dmcrlh* dMractarlstlct of an lixiivl^)'* 
wrlttan raspemw t» • Rron^. CiMract«i-fstf« Awcrilwd IncliMl* contwit 
(iilBM r«pr««tnt»d) mi font (lanviasB ia««t and mchanlcs). 

• TfM VI- i ting SM^pla la not Intanded to ka a tlme4 activity. 
Vlitita r«aawwl»1« Halts, aach aMMlnaa *htm\a tm «l«ait anough 
tiaa Catoat IS-2$ Mtnutas) t« canplata tka II 



• AftSMar Sliaat* and Sarwy BMltlatB atouM aot ba aval labia to 
aMMlnaaa ^rlag tha adtelalatratlen of tkaHrltlng Saaple. 

• SUtrlte«i to aadi aM«laaa tha alngla skaot narfcad MITINS 
SAnm.E (Kara A (M- I) and a thaat of llnad papar. 

o Nm« aaMlMMt ra^ tlia laforaatlon In tba tea teloM ttm Mirdi 
NOTE TO TME Stl»EIIT. CaMliiMa ahoald ba abla to raad indapan- 
daatly aad fellflM tlw diraetlora for tha wlttns M«pla. Ba 
tara aach <uiaiala«a haa tirlttan har mr his naaa on tlw llnad 
PW- 

o Aftar aaaolnaas finish taritlng. collMt tha ftlngla shaat of 
dIrKtlons tpr tha IRITlMi SAf»U md tha actaal tn-ltlnfl sai^la. 

Dlractlons for OasferlWno th« ifrltine lawola (Parw A and B) 

Itechars ar« aakad to dascriha tha Mrttlng saapla tditch studaats 
coa^larad en tha llnad psfar. 6iild«1laas for daftcrlblng tha writins 
SMWl* •rm prasan»ad balair. YOu »hwild os a studants' onwwr shaats 
and fill In tlw ssw:»« »r«clo»id In the boa naar tha right hottam corner 

Tha guidas baloM (forsw A and B) fcserlba ttudant parforwanca for 
savaral charKtarlsttcs. Knfnr to tha guidas to fill In tha bul^ls 
MhoM dascription coaas closest to tha stodant's parfm^amM for awh 
charactaristic. A ona-paga Stawory of floscrlptlaiw folloMS tha Caida 
for aach for*. Tha Suamry can slivlify tha task of dascribing 
charaetaristics. 
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fmm 8 ttfwyy of ftttcriptiw. P1w« the ttitifMf Mftwr 
tliMt mat to thU «i#9» of tfw dlfKC ions , to help you fill In the 
MMiBf ftliMt sickly. 



111. 


$M|»1« SiM 


(A) 


(•) 


(0 


(B) 








At* I ft 

71111 


3/* 


Mil 


\/k or 


112. 


rrokim %tmtm4 


(A) 

ni« 


(!) 
Ito 






- 


HJ 


Action 9tot^ 


(A) 
Yw 


(B) 
Hd 










PrtciM Impiofft 


(A) 
Vm 


(B) 
Mo 










Poll to lOfifMgO 


(A) 
Y«B 


(B) 
Nd 








116. 


Roto toil MfltOflOlft 


(A) 
Vm 


(B) 
Ito 








117. 


Errors In grmmmr 

»4 iKOOploto tontmcM 


(A) 

0-1 


(B) 
2 


(c) 
3 


% 


(c) 
5* 


IIS. 


Errors In o^ltoltntlon 
mtvA pmicttiotlon 


(A) 
0-1 


(B) 
2 


(0 
3 


(0) 
4 


(E) 
5* 


lis. 


Errors In opolllnf 


(A) 


(B) 




(P) 


U) 






O-t 


2 


3 




5* 



Observe that the revised prompt above was written to discourage students 
from regurgitating the prompt Itself; the prompt was designed so that 
responses would be more specific. Hote too that students are guided to 
use precise language ("exact words") and business-appropriate langua^. 
The directions for a*»inlsterlng the assessment now called for description 
rather than evaluation , and the structure for describing performance was 
built Into the scoring guidelines ( Yes , Mo for content criteria). In this 
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May. we hoped teachers would be freed from autanatlcally tnlnking of 
students' writing as Good, Acceptable, and Unacceptable, and instead would 
attend closely to descriptions of Individual criteria as they ap pear In 
the guidelines . The new first criterion (Estimates size of writing sasiple.) 
was Include as a yardstick for describing elanents of form (gramnar and 
complete sentences, capitalization and punctuation, and spelling). The 
new form criteria, we reasoned, would give readers an easier, method of 
describing students* performance for these el^^ts. I.e., counting. 
Together, descriptions of criterion one and the form criteria would yield 
data that were more sensible, data that would consider nmber of errors 
per estimated unit of writing. A program for processing these data could 
be easily set to a standard specified by a district, school, or teacher. 
For example, a passing grade in spelling for a full page or more of writing 
could be four or fewer errors; for about 3A page, three or fewer; about 1/2 
page, two of fewer; about \/h page or less, only none or one error would be 
passing. Further, iwtes at the bottom of descriptions for form criteria 
would, we hoped, clarify guidelines for describing troublesopw situations 
such as run-on sentences. The Summary of Descriptions was designed to 
facilitate recording readers' Judgments. The next step was to field-test 
these revised materials. 

Twenty-four high sctool juniors and seniors who hadn't passed a 
suburban school district's writing senile test were admlnlsterwl the TEPS, 
English Literacy, Form B Writing Senile. All but a few students were 
attending a Basic Writing Skills class; a handful were In an ESL program. 
We have no data that further describe the population for this field test. 
Students' names, where given, obviously were removed from ihcir papers; 
thus, we do not have Information even on students' gender. The English 
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OepartRient Chair and three other English teachers served as readers.* 
T*«> SWRL staff unfamllfar with both the revised and the original materials 
also were readers. Following the procedures of the earlier field tests. 
Me gathered data on students' performance and on estlirated Interrater 
reliability (teachers and SWftt staff), and reviewed comments submitted by 
the sU readers. Note first the following Indication of students' 
performance by criterion. 



Sunaary of Students' Perforwance 
By Critarion and Rater 



Rating 



Criterion 


*1 












r4 


M 


R2 


Si 






M 


sa 






R2 






1. 


3 




1 


2 


1 


t 




2 


7 


7 


5 




13 


12 


11 


11 










2. 


li 


il 


9 


8 


8 


13 


(2 


13 


























3. 


17 


15 


1% 


17 


7 


9 


7 


k 


























k. 


Z 


6 


3 


k 


22 


18 


18 


17 


























s. 


21 


V 


20 


19 


3 


3 


1 


2 


























<. 


20 


19 


12 


18 


k 




9 


3 




















1* 






7. 


1 






S 


6 


3 


3 


1 


1 


1 


3 


h 


2 


0 


2 


2 




16 


9 


9 


8. 


8 


6 


9 


10 


3 


« 


5 


1^ 


1 


k 


2 


3 


k 


0 


3 


1 


8 


8 


2 


3 


9. 


13 


13 


!! 


7 


1 




3 


S 


i 


5 


3 


2 


3 


3 


0 


5 


1 


2 


k 


2 



AThe four teachers divided students' papers so that each pair of 

teachers read half the papers. For computations, the four teachers were 

treated as t%« raters. 

^Raters I and 2 are high school English teachers. Raters 3 and *i 

are SWRL staff. , ^ xu a 

bAn absence of data under a rating Indicates a rating not described 

for that criterion. ^ j * 

cRaters 3 and ^ omlttwJ ratings for three students and commented that 
the three papers were unscorable; hence the total numbers of ratings for 

raters 3 and k are 21 . ... 

<*Ratlng given docs not correspond to any rating described for criterion 
sU. It looks like an error In marking the Answer Sheet. 
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An ejiamlnatlon of the data above sIkms rm major rating |»ttern across 
criteria for the -teachers or ttm SWRL staff. Agreement het%«en ratings 
of the pairs of teachers and |wlr of SWRL .staff is generally respectable 
except for criterion six, as shCMi below. 

SuRMry of Eitfmted int«rr«tcr Itell^lllty 
' By Critsrion tetwmi Tuo Wm\r% of Haters 
(PMrwn Corr«l«tim Coefficients} 





Raters* 




Criterion 


1 t 2 




1. 


.9SS5 


.9^ 


2. 




.5095 


3. 


.azM 


.6860 


k. 


.S222 


.M16 


5. 


1.0000 


.6^2 


6. 


.3508 


.196% 


7. 




,9632 


6. 


.7712 


.4109 


9. 




.8412 



The low agreenent between raters on criterion six warranted close 
attention to readers' r^aariis, as did the lukewarm agreement on cri- 
terion two, four, and eight. Unfortunately, readers did not comiient 
akiout criterion sU. So I asked the SWU. readers to discuss their 
interpretation of the criterion and the way they described this char- 
acteristic of students' performance (the teachers were unavailable). 
The written comments received, recommendations for modifying criteria 
two, four, and eight, and the gist of the discussions with Svmt staff 
concerning criterion six and suggestions for dealing with this criterloi 

^Raters i and 2 are high school English teachers (N » 2*i). Raters 

3 and k are SWRl staff (N » 21). 
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were used In ftcxt steps toward the final version of the TEPS writing 
sffi^le assessoMts. 

The degree of agrement bet««een readers and their conwwnts about cri- 
terion two recommended adjustincnt In the assessment materials. The question 
is •'What gets changed?" The statements and descriptions of each criterion 
are only as productive as students' Interpretation of the pron^jt that stim- 
ulates their writing and readers'^ Interpretation of scoring materials. So 
the question Is raally twofold: Do we change the ^rording of the prompt?, 
or Do we change the wording of the statement and description of the cri- 
terion? A tentative answer lies In readers' conments—ln those remarks 
that address the prompt In general, as well as In those that relate to 
criterion two specifically. A reasonable modification would be In the 
««ording of the prompt, something along the lines of a comment made by 
one of the English teachers, i.e., 

Choose a specific Item(s) that had been ordered. 

Tell specifically what was wrong with the item(s) sent and why It 
should be chaftged: name problem and change. 

The e^»pectatlon Is that readers will be able to describe students' state- 
menl- of the problem more easily If greater specificity Is Induced In the 
writing sample. 

Criterion four might also be Improved with a reworded prompt. Even 
though SWRt staff described the papers similarly for this criterion (the 
coefficient %*as .8416), the teachers were not In such close agreement 
(.5222). And comments from both sets of readers Indicated that this 
criterion was troublesome. A tentative recommendation is to observe the 
results after the pron^t Is rewbrcted to inform the student clearly that 
specifiv details should be lo^rovlsed. 
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Based on imeven and sonei^t low coefficients of estimated tnterrater 
reliability, the third criterion that seened to warrant special attention 
was eight (i^reeaent between teachers, .7712; but between SVmi stdff. 
only >109}. Simi staff did not oofment on this criterion, and the only 
remark nade by teachers was brief: "Run-ons fit here." An examination 
of the ratings studmt by student revealed a curious phenomenon: when 
the pair of raters ^i^d chi criterion eight (10 out of 21 scorable 
pa,«ers), the ispst frequent rating given (7 papers) »«s (A) none or I 
[cap I tall tat I on and punctuation error]; of the other three cases of 
agre^aent, two were (B) 2 (capital I zatlcMi and punctuation errors] and 
only of» was (E) 5 or wore . In other words, raters *fere wore likely to 
agree when they described papers as having few capitalization and punc- 
tuation errors. Falling to locate sources of disagreement by examining 
the papers themselves, I asked ttw two SWRL raters to discuss their 
ratings. The meetif^ bet%#m the three of us *#as Informative. It dls- 
closed an occasional oversight of tim note that told raters not to count 
as errors the points «^re a run-^ sentence stould have been segmented 
(and, thus capitalized). The meeting also enabled us to see the need for 
informing raters whether the letter's acMress, greeting, and closing 
were to be included In the assessable writ Ii^i (they weren't, since these 
were provided In the proo^t). Three other outcomes of the meeting were 
reconmendatlons to advise' raters tt»t the same capitalization or punc 
^ tuatlon error should be counted only once, that sentence fra^nts 

Introdtsced by a capital should not be counted as errors (they'd already 
be counted under criterion seven), and to provide ex^i^les of capitali- 
zation and punctuation errors that could be overlooked easily (and were 
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by the SWfU. raters), for example, capitals within Mords, hyphens, 
apostrophes, and terminal periods. 

Criterion six (Enclttdes tairelated sentences) yielded the lowest 
rating correlations amg ail criteria, both for teachers and for SVIRL 
staff (.3508 «Bid .196*. respectively). Yet no reader commented about It. 
Examining studoits* perfonaances and actual papers and talking Individually 
with the SWRL readers (the teachers were unavailable). I learned that two 
widely different Interpretations of the criterion probably led to Sting 
discrepancies. One reader held a literal Interpretailcm of the word 
refer In the description of the criterion^ Ail of most sentences refer to 
situation described in DIRECTIOHS . That Is, If a sentence did not Include 
words that were specific to the prompt, that sentence was counted among the 
ones that didn't Meet the descrlptIon--tl»y didn't refer to the situation. 
The other reader Interpreted refer wore loosely, essentially as "having 
to do with." For exaR^le, a pi^per Included these sentences: "The ship- 
ment we recfeyed tsic] was of no use to us. It tsld was for the 
Industrial building next door from us." Another paper had this sentence: 
"This may have come from one of your employees wrcmg." One reader Judged 
that these sentences did not refer to the situation described In the 
pron^t. The Other reader, however, thought the sentences did refer to - . 
the general situation of a letter of request for an order adjustment. 
Similarly, redder viewed as tmrelated to the prompt those sentences 
that expressed «aenltles of business correspwidcncc; the other reader 
thought such sentences were related, for instance. "I am very sorry to 
write this kind of letter" and •♦Thank you for helping [sic]." 

To eliminate the confusion that might arise from the negative 
criterion statement Excludes unrelated sentences . It was changed tc^ Stays 
on topic. Also to reflect coominlcatlon that crosses sentence boundaries 



48 

and to avoid the wide Interpretation typically given to the word most 
Uttoe people thti^i of aoftt ae 51 percent or more, others think of It 
as closer to 80 percent or so), the criterion description was reworded* as 
Senarally relates to situation descrlhed In OIRECTIOWS (this revision 
would also be expected to eliminate the problems of Interpretation 
associated wItH the word refer ). Another recoinraendatlon was to include 
a note In the scoring guidelines that amenities of business correspondence 
do not constitute abandonment of the topic. 1 had some concern that If 
the suggestions above were Incorporated In a revision, the resulting 
rating would be Yes for all or nearly alT papers, since papers that were 
truly irrelevant to the topic were to be left unscored. A criterion 
that provides consistently uniform ratings would be wasteful . Why not 
Just describe the performame without rating papers for It? But this 
field test demonstrated that teachers nsiy not Jwlge a paper unscorable 
as readily as other raters (I.e., SlfRL staff) would. In fact, three 
papers Judged unscorable by SWm. staff, for content Irrelevant to the 
prompt, were Indeed scored by teachers. Could the reason be that 
teachers are In need of Information concerning their m students' 
writing performance «nd are thus more Inclined to retain a paper for 
further scoring? In administrations of the assessment In Its final 
form, teachers wlU be the readers and raters. Hme, this line of 
reasoning advised suspending, at least for now, the concern regarding 
uniform rating on criterion sU. 

Although we would tlice to think that the assessments are "final." 
that would be premature. Both writing sample assessments will be 
subject to further revision based on additional experience with them. 
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As with any written coffimiinUation, a writing asse&sment can be 
dashed off in one draft and it can also he revised to death. Reasonable 
effort is between those extremes. Reader reaction is the best gauge of 
«diat to do after the first draft. Because the "readers" of a writing 
assessment react In writing, the ruction Is unambiguous. All the 
"author" has to do Is to respond to the reactions that are registered. 
Iteinft^. that Is stralghtforwa^* but It's not as simple as It seems. 

id. 
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